In the field of digital image processing and display, often there is a need to display an image in high resolution based on a low-resolution image sources. For example, in the IPTV (Internet Protocol Television) application, the streaming video sources may be transmitted at a resolution much lower than the native resolution of a television set. In order to display the low-resolution streaming video contents on the television set, the source video has to be up converted to the full native resolution supported by the television set. Similar requirement of high-resolution image reconstruction has also been noted in digital camera, satellite remote sensing imaging, and medical imaging.
Various high-resolution image reconstruction techniques have been reported in the literature. One category of high-resolution image reconstruction techniques is based on a single low-resolution frame. Bilinear interpolation is one of the popular conventional approaches to high-resolution image reconstruction. Bilinear interpolation performs linear interpolation in both horizontal and vertical directions. The computation associated with bilinear interpolation is relatively simple and the visual quality of the interpolated image is usually acceptable. Nevertheless, the bilinear interpolation process often causes smoothness and other artifacts around object edges due to interpolation in the direction perpendicular to the edge. Accordingly, various modified interpolation techniques based on single frame have been reported in the literature to enhance sharpness around object edges. For example, an edge-directed interpolation is disclosed by Li, et al. (“New Edge-Directed Interpolation”, in IEEE TRANSACTIONS ON IMAGE PROCESSING, pp. 1521-1527, VOL. 10, NO. 10, OCTOBER 2001) and the method is termed as NEDI in this disclosure. The NEDI method utilizes the edge-directed property of covariance-based adaptation instead of explicitly estimating the edge orientation. The NEDI method has been shown to produce improved quality over the bilinear interpolation method.
For more advanced high-resolution image reconstruction, signal processing techniques are used to generate the high-resolution image using multiple low-resolution images. The multi-frame based high-resolution image reconstruction is also referred to as super-resolution image reconstruction in the field. In this disclosure, super-resolution image reconstruction and high-resolution image reconstruction may be used interchangeably. The multiple images may contain moving objects in the scene and motion estimation has to be used to match corresponding pixels among different frames. The estimation of motion information is referred to as registration in the field. In typical multi-frame based high-resolution image reconstruction, three stages of processing are involved, as shown in FIG. 1, including registration 110, interpolation 120 and restoration 130. The stage of registration 110 receives multiple low-resolution images, yi, i=1, . . . , P. The outputs from registration 110 are motion-compensated low-resolution images. The stage of interpolation 120 reconstructs the high-resolution image on a high-resolution grid using the motion-compensated low-resolution images. There are various interpolation techniques used in the field. The stage of restoration 130 is used to reduce the noise and/or artifact introduced in the interpolation stage. An overview of super-resolution image reconstruction is described by Park et al. (Super-Resolution Image Reconstruction: A Technical Overview, IEEE SIGNAL PROCESSING MAGAZINE, pp. 21-36, May 2003).
Recently, kernel regression has been introduced by Takeda et al. (“Kernel Regression for Image Processing and Reconstruction”, IEEE TRANSACTIONS ON IMAGE PROCESSING, pp. 349-366, VOL. 16, NO. 2, FEBRUARY 2007) for high-resolution image reconstruction. For 2-D kernel regression, a two-dimensional regression model is used to describe the observed data yi at location xi:yi=ƒ(xi)+εi, i=1, . . . , P, xi=(x1i,x2i),  (1)where (x1i,x2i) is the coordinate in the two dimensional space, ƒ(·) is the regression function to be determined, and εi is an independent and identically distributed zero mean noise. P represents the number of low-resolution data samples in the region of interest used to construct a high-resolution sample. The specific form of ƒ(xi) may be unspecified. However, it can be assumed that the regression function is locally smooth around location x where the high-resolution sample will be reconstructed. Accordingly, the regression function at location xi can be represented using a form of Taylor series:ƒ(xi)=β0+β1T(xi−x)+β2T vech{(xi−x)(xi−x)T}+ . . .  (2)where β0=ƒ(x), β1 and β2 can be represented as:
                                          β            1                    =                                    ∇                              f                ⁡                                  (                  x                  )                                                      =                                          [                                                                            ∂                                              f                        ⁡                                                  (                          x                          )                                                                                                            ∂                                              x                        1                                                                              ,                                                            ∂                                              f                        ⁡                                                  (                          x                          )                                                                                                            ∂                                              x                        2                                                                                            ]                            T                                      ,        and                            (                  3          ⁢          a                )                                          β          2                =                                                            1                2                            ⁡                              [                                                                                                    ∂                        2                                            ⁢                                              f                        ⁡                                                  (                          x                          )                                                                                                            ∂                                              x                        1                        2                                                                              ,                                                                                    ∂                        2                                            ⁢                                              f                        ⁡                                                  (                          x                          )                                                                                                                                    ∂                                                  x                          1                                                                    ⁢                                              ∂                                                  x                          2                                                                                                      ,                                                                                    ∂                        2                                            ⁢                                              f                        ⁡                                                  (                          x                          )                                                                                                            ∂                                              x                        2                        2                                                                                            ]                                      T                    .                                    (                  3          ⁢          b                )            Term vech(·) is defined as the half-vectorization operator of the “lower-triangular” portion of a symmetric matrix, e.g.,
                              vech          ⁡                      (                                                            a                                                  b                                                  c                                                                              b                                                  e                                                  f                                                                              c                                                  f                                                  i                                                      )                          =                                            [                              a                ⁢                                                                  ⁢                b                ⁢                                                                  ⁢                c                ⁢                                                                  ⁢                e                ⁢                                                                  ⁢                f                ⁢                                                                  ⁢                i                            ]                        T                    .                                    (        4        )            
Base on equation (2), the regression function can be specified if all βn's are known. Accordingly, the high-resolution reconstruction problem for estimating the high-resolution construction at x based on observed data yi, i=0, . . . , P, can be solved by determining the coefficient set, {βn}. One approach to determining the coefficient set of {βn} is to solve the following optimization problem:
                              min                      {                          β              n                        }                          ⁢                                            ∑                                                                                                    i              =              0                        N                    [                                          ⁢                                                                 y                i                            -                              β                0                            -                                                β                  1                  T                                ⁡                                  (                                                            x                      i                                        -                    x                                    )                                            -                                                β                  2                  T                                ⁢                vech                ⁢                                                      {                                                                                            (                                                                                    x                              i                                                        -                            x                                                    )                                                ⁢                                                                              (                                                                                          x                                i                                                            -                              x                                                        )                                                    T                                                                    -                      …                                        ]                                    2                                ⁢                                                                            K                      H                                        ⁡                                          (                                                                        x                          i                                                -                        x                                            )                                                        .                                                                                        (        5        )            KH(·) in equation (5) is the kernel function which penalizes distance away from the local position where the approximation is centered. The method of high-resolution reconstruction based on a regression model is termed as kernel regression.
To further improve the performance of high-resolution image reconstruction based on kernel regression, Takeda et al. disclosed data-adaptive kernel regression. Data-adapted kernel regression methods takes into consideration of the radiometric properties of these samples in addition to the sample location and density. Therefore, the effective size and shape of the regression kernel are adapted locally to image features such as edges. Takeda et al., incorporates a feature of kernel function associated with the measured data that implicitly measures a function of the local gradient estimated between neighboring values and to use this estimate to weight the respective measurements. Accordingly, a two-dimensional steering kernel function is introduced by Takeda et al. and the two-dimensional steering kernel function has a form shown below:
                                                        K              H                        ⁡                          (                                                x                  i                                -                x                            )                                =                                                                      det                  ⁡                                      (                                          C                      i                                        )                                                                              2                ⁢                π                                      ⁢            exp            ⁢                          {                              -                                                                                                    (                                                                              x                            i                                                    -                          x                                                )                                            T                                        ⁢                                                                  C                        i                                            ⁡                                              (                                                                              x                            i                                                    -                          x                                                )                                                                                                  2                    ⁢                    h                                                              }                                      ,                            (        6        )            where a Gaussian kernel is used, h is the smoothing parameter to control the strength of the penalty, and Ci is covariance matrices based on differences in the local gray-values at xi.
The steering kernel in equation (6) takes into consideration of local data characteristics, such as edges. Takeda et al. have reported performance improvement over bilinear interpolation and classic kernel regression. Nevertheless, the steering kernel in (6) is agnostic of object motion within the multiple frames and consequently the performance may be compromised. In a subsequent work by Takeda et al., a three dimensional steering kernel (“Spatio-Temporal Video Interpolation Using Motion-Assisted Steering Kernel (MASK) Regression”, Proceedings of IEEE International Conference on Image Processing, pp. 637-640, 12-15 Oct. 2008) is disclosed. The spatio-temporal three-dimensional kernel is termed as MASK by Takeda et al. The adaptive spatio-temporal steering kernel called motion-assisted steering kernel (MASK) is shown in equation (7):
                                          K            MASK                    ≡                                    1                              det                ⁡                                  (                                      H                    i                    S                                    )                                                      ⁢                          K              ⁡                              (                                                                            (                                              H                        i                        S                                            )                                                              -                      1                                                        ⁢                                                            H                      i                      m                                        ⁡                                          (                                                                        x                          i                                                -                        x                                            )                                                                      )                                      ⁢                                          K                                  h                  i                  t                                            ⁡                              (                                                      t                    i                                    -                  t                                )                                                    ,                            (        7        )            where HiS is a 3×3 spatial steering matrix, Him is a 3×3 motion steering matrix, and hit is a temporal steering parameter. The MASK method addresses the motion factor by introducing 3×3 motion steering matrix, Him. It is well known that the motion-compensated residues are usually more prominent in the area undergoing complex motion. The pixels from the areas undergoing complex motion may contribute significantly to the overall residue calculation during the optimization process. Nevertheless, the accuracy of registration for areas with complex motion is not taken into consideration in the MASK approach. Consequently, the performance of the MASK-based approach may suffer performance degradation when complex motion is involved in the multiple frames. In additional, the MASK method lacks the capability to handle rotational object motion in the multiple frames. Therefore, neither the two-dimensional kernel regression nor the three-dimensional based MASK method can adequately address complex motion in the multiple frames. It is desirable to develop super-resolution image construction that can deliver improved performance and handle complex motion in the multiple frames.